One way to summarize the core philosophy of science-based medicine (SBM) is that we favor raising both the ceiling and the floor on the quality of scientific evidence used to inform our healthcare decision making. Mainstream medicine endeavors to be scientific, but that has always been an aspirational goal. Doing rigorous science is hard, and applying it to something as complex and varied as medicine is even harder. There are countless pitfalls, many explored here at SBM. While we are trying to increase the overall quality of science in medicine, we pay particular attention to the low end of the spectrum, where the abused and errors are more flagrant. There are also those who are attacking the scientific basis for medicine in principle, and we feel it is especially important to take on such attacks vigorously.
In many ways the challenges to SBM has been increasing, standards have weakened, and regulations watered down. Ironically it seems that while the ceiling of high quality has been constantly and incrementally rising, the floor has fallen out beneath us. We obsess over nuances like where to set the P-value and the effects of citation bias, while at the bottom, rank magic and pseudoscience is gaining a foothold. Today I am going to write about both ends of the spectrum at once, highlighting what has arguably been one of the best trends in scientific medicine in the last two decades – preregistering clinical trials.
This idea was first floated in 1986, but gained steam around the turn of the millennium. The first US federal law requiring trial registration was in 1997, the Food and Drug Administration Modernization Act of 1997 (FDAMA). This act included a provision requiring clinical drug trials approved under the investigational new drug application to register their trials with the government. In 2000 the NIH created the clinicaltrials.gov website, providing a place for companies to register their trials. Since then the benefits of pre-registration have become increasingly clear.
Prior to preregistration, most scientific studies were conducted essentially in secret. Only the people directly involved in the research or its funding were likely to be aware of it. This meant that there was not way to ensure that the study was conducted as originally planned, or that the results would ever be published. Pharmaceutical companies, for example, could theoretically conduct five studies looking at the efficacy of their drug, and only publish the two that showed a positive effect while sweeping the other three under the rug. They even argued that they owned the data of the trials that they funded and conducted, so they could do with it what they wanted. They could even hide studies by conducting them overseas. This was the initial impetus for registering trials.
But this is just a slice of a systemic problem within science, called publication bias or the file-drawer effect. Studies that show more positive effects are more likely to ever be published, are published earlier, and are cited more often. If we perform a systematic review of published studies, therefore, we are seeing a biased sample, which distorts the answer. Publication bias alone can make a worthless treatment seem effective.
A more subtle problem with clinical trials is P-hacking, which we have discussed previously on SBM. In brief, P-hacking may occur whenever the details of how a study is conducted and the data analyzed are determined after results are seen. This creates the potential, whether conscious or not, of tweaking the methods in order to produce statistically significant results. It’s actually not hard to do – let’s just collect another 20 samples and then reanalyze the data. Deciding when to stop collecting data, which outcome measures or comparisons to use, and which statistical analysis to use can all alter the significance of the outcome. Preregistering trials prevents P-hacking, because you have to spell out the exact methods that will be used. At least, if the researchers change the methods mid-stream, it will be obvious.
Preregistering clinical trials is therefore an extremely effective method of improving the quality of scientific data. However, in order to fully realize the benefits of registration we need to completely close the loop – we need to include in any systematic review an analysis of which studies were preregistered and if any alterations were made post registration. A recent study looking at homeopathy trials illustrates this.
Homeopathy, for any readers who may be unaware, is 100% pure pseudoscience. Homeopathic potions are essentially magic water, in which fanciful substances are diluted out of existence in the hopes that their “essence” will remain. As medicine, they are worthless, and a century of research shows this to be true. Because there is so much research into homeopathy, however, this does create a useful opportunity for SBM. We can ask – what does the medical literature look like when it involves an intervention that we know for certain (as much certainty as we can have in science) has no effect? The homeopathic literature gives us a detailed answer, and now we have one more example to add to the list.
A 2022 BMJ study looked at homeopathy clinical trials with respect to their registration status. They found:
Since 2002, almost 38% of registered homeopathy trials have remained unpublished, and 50% of published randomised controlled trials (RCTs) have not been registered. Retrospective registration was more common than prospective registration. Furthermore, 25% of primary outcomes were altered or changed compared with the registry. Although we could detect a statistically significant trend toward an increase of registrations of homeopathy trials (p=0.001), almost 30% of RCTs published during the past 5 years had not been registered.
More than a third of registered trials were never published, and half of published trials were never registered. Further, most registered trials were not preregistered, but only after the fact. Finally, in 25% of the registered trials significant changes were made to the methods after registration. Essentially, the homeopaths are doing it wrong, undermining the entire point of preregistering trials. But, the fact that some trials are registered does give us the opportunity to explore the effect of registration:
A meta-analysis stratified by registration status of RCTs revealed substantially larger treatment effects of unregistered RCTs (SMD: −0.53, 95% CI −0.87 to −0.20) than registered RCTs (SMD: −0.14, 95% CI −0.35 to 0.07).
As predicted – unregistered trials show a larger effect, likely because of publication bias and P-hacking. The effect would likely be even more dramatic if only preregistered (not post-registered) trials were included, and only those that did not change their methods from what was registered. What does all this mean?
Again, it is consistent with the fact that homeopathy is worthless and its potions ineffective. More importantly this has implications for all clinical research. It is perfectly reasonable for medical journals to require preregistration for any study they publish. Further, no systematic review should be considered complete if it does not consider the registration status of each study it examines. Registration does not mean anything if we don’t use the data – closing the loop. If this becomes not only standard but universal, then we will realize the full benefits of an idea first proposed 36 years ago.
